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Executive summary 


The project 


The Vocabulary Enrichment Full Programme combined three existing programmes — the Vocabulary 
Enrichment Intervention Programme (VEIP), Sounds-Write and Literacy Plus — and aimed to improve 
the reading abilities of pupils in Year 7. VEIP is a structured scheme that teaches children new words 
and encourages them to use these words in speaking and writing; Sounds-Write is a phonics 
programme; and Literacy Plus targets pupils at Key Stage 3 who are behind with their literacy skills 
and provides additional support to move them from Level 3 to Level 4 in English. 


Bolton Local Authority devised the full programme and supported its delivery in twelve local schools. 
Schools were encouraged to use the three elements of the programme to build their own scheme of 
work. Existing class teachers in the schools received training on all three elements and delivered the 
intervention to Year 7 pupils who had not reached a secure Level 4 in English at the end of Key Stage 
2. The intervention replaced these pupils’ usual English lessons for three consecutive half terms 
between September 2013 and February 2014 (approximately 19 teaching weeks). The frequency and 
length of lessons depended on the schools’ individual timetabling arrangements. 


The programme was evaluated using a randomised controlled trial, which compared the intervention 
to a ‘business-as-usual’ control group. As the teachers delivering the programme had supervision and 
support from the development team, this evaluation is an efficacy trial. 


The delivery and evaluation of this project was funded by the Education Endowment Foundation as 
one of 23 projects focused on literacy catch-up at the transition from primary to secondary school. It 
was one of four projects funded with a particular focus on reading comprehension. 


1. This evaluation does not provide any evidence that the Vocabulary Enrichment Full 
Programme had an impact on attainment in reading. 


2. There was considerable variation in the way that the three components of the intervention were 
combined. The intervention would therefore have benefited from a pilot stage to refine the 
elements used in the Full Programme. 


3. Pupils generally appeared to engage well with the intervention. However, some higher-ability 
pupils found the programme too easy and teachers felt that they should not be in the study. 


4. In general, Literacy Plus and Sounds-Write were perceived by teachers to be less appropriate 
or less essential than VEIP. Sounds-Write is aimed at a younger audience than Year 7, but 
was used as there is a lack of phonics programmes specifically designed for secondary school 


pupils. 


5. Given the emphasis placed by some teachers on vocabulary skills, a future trial might use a 
different test that captures this aspect better than the New Group Reading Test. 
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What impact did it have? 


There is no evidence from this evaluation that the Vocabulary Enrichment Full Programme had an 
impact on pupils’ reading ability. The headline findings suggest that on average the programme had a 
slightly positive impact on the reading ability across all pupils and across pupils eligible for free school 
meals, with effect sizes of 0.06 and 0.01, respectively. However, these estimates are not statistically 
significant so we are not able to conclude that the observed effects were caused by the programme 
rather than occurring by chance. 


Observations and interviews with staff suggested that pupils generally appeared to engage well with 
the intervention. However, some higher-ability pupils found the programme too easy and teachers felt 
that they should not be in the study. 


Vocabulary 0.06 

Enrichment Full 282 0 06 0.18 +1 No 4488 ££ 
Programme (-0.06, 0.18) 
Vocabulary 0.01 


Enrichment Full 
Programme (FSM 


pupils) 


81 — (-0.23, 0.25) 0 No 


*Effect sizes with confidence intervals that pass through 0 are not ‘statistically significant’, so we do not have sufficient evidence to conclude 
that the observed effect was caused by the programme rather than occurring by chance. 
*“For more information about evidence ratings, see Appendix IV in the main evaluation report. 
Evidence ratings are not provided for subgroup analyses, which will always be less secure than overall findings 
*“ For more information about cost ratings, see Appendix V in the main evaluation report. 


How secure is this finding? 


Overall, the evaluation findings are rated as considerably secure. This assessment takes into account 
a number of factors including the study’s design, size, and level of drop-out. 


Impact was assessed through a two-arm pupil-randomised controlled trial in twelve schools; 649 
pupils in Year 7 were randomised to either the Vocabulary Enrichment Full Programme or the 
‘business as usual’ control group. Although there was attrition of 12% by the final analysis, there was 
no evidence that this led to bias on observable characteristics that might impinge on the security of the 
findings. 


There had been no assessment of the impact of the Full Programme prior to this trial. The main 
component of the intervention, the Vocabulary Enrichment Intervention Programme (VEIP), was one 
arm of a randomised controlled trial undertaken by its developer in 21 secondary schools in outer 
London areas from 2006. The developer reported a significant impact of VEIP on vocabulary test 
scores, although the trial has not been reported in full. There is a lack of published evidence for the 
effectiveness of Literacy Plus and Sounds-Write. 


As the present study represents the first formal evaluation of the Full Programme, and teachers had 
supervision and support from the development team and from a visiting educational psychologist, this 
study can be regarded as an efficacy trial. Efficacy trials seek to test evaluations in the best possible 
conditions to see if they hold promise. They do not indicate the extent to which the intervention will be 
effective in all schools, because the participating schools are selected from one area and the 
programme is closely managed by the developers. 
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The primary outcome was reading ability as assessed by scores from the GL Assessment New Group 
Reading Test (NGRT). The secondary outcomes were the two NGRT subscales: sentence completion 
and passage comprehension. Test administration was carried out by staff at the participating schools. 
Efforts were made to ensure that test administration was blind in every school, although complete 
blindness, as if delivered externally, could not be guaranteed. The test marking was carried out by GL 
Assessment and was therefore blind. 


Analysis was completed on an ‘intention to treat’ basis, reflecting the reality of how interventions are 
delivered in practice. 


How much does it cost? 


The project had core costs, such as the externally-delivered training and teaching resources, and 
variable (additional) costs, such as the training delivered by the developers and ongoing support from 
an educational psychologist. The minimum (core) cost per school to train and equip four teachers to 
deliver the intervention to 100 pupils would be around £7,470 (£75 per pupil). The maximum cost (core 
plus variable costs) per school could be in the region of £10,990 (£110 per pupil). 


Salary costs £56 (cover costs related to teacher training) 

Training £27.40 

Resources £18.50 (photocopying, stationery and programme materials) 
Other £8 (support from educational psychologist) 
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1. Introduction 


1.1 Intervention 


The Vocabulary Enrichment Full Programme (VEFP) is a combination of the Vocabulary Enrichment 
Intervention Programme (VEIP, developed by Dr Victoria Joffe), the Literacy Plus scheme (developed 
by a previous UK government), and the Sounds-Write scheme (developed by qualified teachers Susan 
Case, David Philpot and John Walker). The three schemes differ from one another in terms of the 
emphasis placed on the development of reading, writing, speaking and listening skills. The combined 
intervention aimed to emphasise the development of literacy skills, in particular reading 
comprehension (the outcome measure). Schools were encouraged to use the three elements, with 
VEIP being the core element, to build their own scheme of work for low-attaining Year 7 pupils who 
were randomly selected to participate in the trial. 


1.2 Background evidence 


VEFP is a new scheme devised for the purposes of this trial by Bolton Local Authority (including 
members of its Educational Psychology Service, Education Improvement Team, and Paediatric 
Speech and Language Therapy Team) and a school-based literacy consultant. This intervention 
should be seen as being at the efficacy trial stage of development. Efficacy trials test whether an 
intervention can work under ideal or developer-led conditions in a large number (e.g. ten or more) of 
schools. This evaluation was funded by the Education Endowment Foundation, as one of 23 projects 
focused on literacy catch-up projects at the primary-secondary transition. It was one of four 
programmes funded with a particular focus on improving reading comprehension. 


Previous to this trial, there had been no formal or informal impact evaluation of this intervention as a 
package (i.e. the Full Programme). Findings from evaluations of its constituent parts are outlined 
below. 


VEIP has previously been tested in a randomised controlled trial in 21 mainstream secondary schools 
in outer London, which was funded by the Nuffield Foundation. This involved 462 pupils aged 12, over 
90% of whom had speech, language and communication difficulties. The programme was delivered by 
teaching assistants (TAs) and learning support assistants (LSAs) with support from speech and 
language therapists. Pupils attended three 45-minute sessions per week for six weeks. The 
programme developer reported a significant impact of VEIP on vocabulary test scores, although the 
evaluator has not been able to source a published full report of the study or its findings. A number of 
other schools in London have since adopted the programme. One school within this Bolton trial used 
VEIP (the main component of the Full Programme) with Year 7 pupils within its Speech and Language 
Therapy unit in 2012-13 in order to identify any delivery challenges and amendments needed to 
facilitate use of this programme during the trial. 


Sounds-Write is an established phonics programme; its developers report that it is used extensively in 
primary and secondary schools throughout the UK and overseas. The developers’ own longitudinal 
evaluation of the programme (an assessment of the progress of Key Stage 1 pupils at 24 schools in 
England) reported that, for pupils who scored below, but within six months of, their actual age level at 
a baseline spelling test in Reception (252 pupils), 91% had age-appropriate scores on the same test 
by Year 2 (Sounds-Write, 2009). This evaluation did not include a comparison group. VEFP’s 
developers found that available phonics programmes tend to be designed for the primary age group. 
Sounds-Write was chosen for this trial as it seemed the most amenable to adaptation for the 
secondary age group. 


There does not appear to be any substantial evaluation evidence of the effectiveness of Literacy Plus. 
The Centre for the Use of Research and Evidence in Education (CUREE) conducted a small 
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qualitative evaluation on boys’ progress in reading and writing (a total of 32 boys in Year 7 and Year 8 
in six schools in England) (CUREE, 2010). This evaluation reported that pupils gained the confidence 
required to engage in literacy activities, and that their reading, writing, speaking and listening skills 
improved. This evaluation did not include a comparison group. 


1.3 Evaluation objectives 


The impact evaluation sought to answer the following research questions: 


6. What is the impact of the Vocabulary Enrichment Full Programme (VEFP) on reading ability? 


7. Are improvements in attainment moderated by National Curriculum reading level or whether a 
pupil receives the pupil premium? 


The purpose of the process evaluation was to assess the intervention in terms of fidelity to the 
programme intentions and the scalability of the programme. 


1.4 Project team 


VEFP was designed by Bolton Local Authority and delivered by teachers at the twelve participating 
schools, with support from educational psychologists employed by Bolton LA. The evaluation team at 
NFER was led by Dr Ben Styles with support from Pippa Lord. Sally Bradshaw carried out the 
statistical analysis. Eleanor Stevens, Rebecca Clarkson and Katherine Fowler carried out the process 
evaluation. 


1.5 Ethical review 


Headteachers consented to the trial being carried out within their schools. This consent was followed 
up by a letter to parents allowing opt-out consent (see Appendix C). The pattern of headteacher 
consent followed by parental opt-out consent, as adopted for other EEF literacy catch-up trials run at 
NFER, was approved by NFER’s Code of Practice Committee on 23 January 2013. 


Trial registration 


This trial has been registered at http://www.controlled-trials.com/ISRCTN67513320/. 
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2. Methodology 


2.1 Design 


The evaluation was run as a randomised controlled trial, with 649 pupils in Year 7 across twelve 
secondary schools, randomised at pupil level to two groups — intervention and a waitlist control — 
within the same school. Pupils in the intervention group were intended to receive the VEFP; pupils in 
the control group experienced their usual English teaching. Pupils were randomised within each 
school and then those in the intervention group were placed into intervention classes, and those in the 
control group received ‘business-as-usual’ English lessons, so intervention and control arms were 
occurring within the same school. The schools recruited had agreed to allocate children to classes 
using the randomisation result and then timetable appropriately. Pupils were tested for reading ability 
both before and after the intervention. This design sought to determine whether the VEFP improved 
reading ability at rates above those of normal classroom teaching. 


Restricting eligibility to Year 7 rather than Year 6 was important to ensure that adequate numbers of 
pupils and staff were available at each school. The developers had designed the intervention for pupils 
predicted to attain Level 4c or below in reading at Key Stage 2, and suggested involving schools in 
another local authority to achieve adequate numbers of pupils randomised for the trial. However, EEF 
made the decision to extend the eligibility criteria to include pupils predicted to achieve Level 4b. This 
trial was originally intended to have two experimental groups: one to be taught the Vocabulary 
Enrichment Full Programme, comprising VEIP, Literacy Plus, Sounds-Write and bridging activities to 
link learning to the wider curriculum/personal reading; the other to receive this programme minus 
VEIP. This design proved logistically difficult to run, largely due to the timetabling constraints in 
operation at each school. At a recruitment event, it became apparent that a three-arm trial was not 
feasible and the protocol was changed to a two-arm trial. The design was changed during school 
recruitment and before memoranda of understanding were signed by schools. 


2.2 Eligibility 


All 17 secondary schools located in Bolton Local Authority were eligible for inclusion in the trial. On 25 
February 2013, a recruitment event was held during which it was explained to school representatives 
what would be required of them. Eleven schools signed up formally after this event. Pupil eligibility 
was based on predicted Key Stage 2 Reading results obtained from feeder primary schools. Pupils 
who were predicted to achieve at or below Level 4b were eligible for the trial. Across the original 11 
recruited schools, 1990 pupils were screened for eligibility by Bolton LA. Those attending primary 
schools from outside Bolton were not screened and therefore not included in the trial. A final school 
joined the project after the pupil eligibility screening had been carried out so did its own initial 
screening. This brought the total number of schools involved in the project to twelve. 


Consent was initially sought from headteachers who had to complete a memorandum of 
understanding (see Appendix B). Opt-out consent was sought from parents of pupils who met the 
eligibility criteria and had been selected for participation (see Appendix C). 


Pupil lists could not be transferred to NFER until opt-out parental consent had been obtained. Ideally, 
this would have resulted in randomisation occurring after pre-testing. However, schools required 
randomisation to happen in advance of autumn term for timetabling reasons. Despite concerted efforts 
on behalf of NFER and Bolton LA to delay randomisation, at least one school threatened to withdraw 
from the trial if randomisation did not happen before September. Randomisation therefore took place 
before baseline testing and before opt-out parental consent was obtained. In one school, the incorrect 
timetable groupings had been sent for the main within-school randomisations. In this school, the 
revised pupil lists were sent after parental consent was obtained and then randomised. A list of 
anonymous pupil identifiers was used for randomisation and this was linked to UPNs and names after 
parental consent had been obtained. Between randomisation and pre-testing, some pupils were 
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excluded from the trial for a variety of reasons: their parents had opted out, or their actual Key Stage 2 
results were too high, or they had left the school, or the parental consent letter was not delivered. All 
exclusions at this stage were made by individuals who did not know whether pupils were allocated to 
intervention or control. 


2.3 Intervention 


The Vocabulary Enrichment Full Programme trialled in this evaluation comprised three core elements: 
the Vocabulary Enrichment Intervention Programme (VEIP), the Sounds-Write scheme, and the 
Literacy Plus scheme. It was delivered in place of schools’ usual Year 7 English curriculum during 
three consecutive half terms by existing class teachers. These teachers had regular supervision and 
support from an educational psychologist based at Bolton Local Authority. The control group in each 
school received ‘business as usual’, i.e. they were taught the school’s usual Year 7 scheme of work 
for English. Intervention pupils were removed from their English classes, thus control pupils were 
taught in smaller classes than under a true ‘business as usual’ scenario. 


The project designers allowed flexibility in the way that teachers approached the design and delivery 
of lessons in each school, so that the intervention would suit the needs of the school, teachers and 
pupils involved. The main stipulation was that the Vocabulary Enrichment Intervention Programme 
should be the principal component. In practice, delivery varied between schools in terms of the 
duration of each lesson (50-60 minutes), the number of lessons per week (three to five), the number of 
pupils per group (ranging from six to 30 at the outset of the trial), and how different elements of the 
three programmes were combined. Teachers focused on Vocabulary Enrichment throughout the 
majority or all of the lessons, as instructed, and used elements of Sounds-Write and Literacy Plus to 
varying extents. It is therefore difficult to assess overall fidelity to the intervention as schools were not 
following a prescriptive combination of programmes and were responsive to the needs of their pupils 
(for instance, by adjusting the pace of delivery). This is explained further under ‘Intervention delivery’ 
in the Process Evaluation section. Each core element of the intervention is described below. 


The VEIP is a structured scheme to teach children new words, and to encourage them to use these 
words in speaking and writing. It seeks to achieve these aims by cultivating children’s enthusiasm for 
words, and teaching them specific strategies for learning and recalling new vocabulary independently 
through a range of techniques, such as categorisation, mind mapping, word-building, and the use of a 
dictionary and thesaurus. Children are taught to recognise the structure of words, and to understand 
characteristics of language such as idiomatic phrases. The programme was devised by Dr Victoria 
Joffe as part of a research project funded by the Nuffield Foundation, and was primarily designed for 
children with communication and language impairment. The theory behind using VEIP as the main 
component of the intervention was that it would improve reading comprehension by: extending 
vocabulary; developing understanding of words in context, such as how sentence construction affects 
meaning and the use of different tenses, plurals etc.; and developing understanding of non-literal 
language. 


The VEIP teaching would be cumulative and sequential; however, teachers were advised to be 
selective about the VEIP material used as it is extensive. Additionally, pupils should learn about 
vocabulary for subject areas other than English: this is an explicit part of the VEIP programme, in 
which ‘Mission to Achieve’ activities link to vocabulary learning in other subjects and in personal 
reading beyond school. 


Sounds-Write is a phonics programme that follows a structured and cumulative sequence of teaching 
and learning. It is primarily for use in Key Stages 1 and 2. Sounds-Write was chosen for VEFP 
because there is a lack of phonics programmes specifically designed for the secondary age range, 
and this programme seemed to be the most easily adaptable to the needs of secondary school pupils. 
This scheme (training and resource materials) has been acknowledged by the Department for 
Education as meeting the core criteria of phonics products that may be used in schools (DfE, 2013). It 
was intended that the Sounds-Write training would ensure that teachers had the understanding of 
phonics necessary to deliver VEIP effectively. Sounds-Write elements were chosen to complement 
VEIP by providing pupils with additional strategies for reading and spelling new words. 
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Literacy Plus, devised by a previous UK government, is a scheme developed for Key Stage 3 lower 
attainers to facilitate progress from Level 3 to Level 4 (DfE, 2011). It is designed to be used for low- 
ability children for at least two periods a week, in addition and complementary to national curriculum 
English. It includes reading, writing, speaking and listening skills, and follows a typical structure of 
starter activity, guided group work, and plenary. Pupils are expected to be supported by an additional 
adult e.g. TA or mentor (in addition to their class teacher). Literacy Plus was chosen for the 
intervention as it was felt to offer reading context that was firmly based in the national curriculum for 
English, and therefore would be a suitable complement to VEIP. Four units were selected from 
Literacy Plus for use in this intervention (My Character, Going for Gold, DIY Writing, and Are We 
There Yet?). 


During the teachers’ training, it was suggested that teaching time over the three half terms could be 
divided into blocks of four lessons which would use the following sequence: 


e Lesson 1 of a block of 4 lessons: exclusive focus on the Vocabulary Enrichment Intervention 
Programme (VEIP) 


e Lessons 2-4 of a block of 4 lessons: Draw on learning from VEIP. Use Sounds-Write 
approaches to learn new vocabulary within VEIP and Literacy Plus. For instance, Sounds- 
Write could be used as a starter activity, series of whole-class short activities to introduce new 
vocabulary from VEIP, or with small groups (with TA support). The teacher would also use 
their own resources and teaching skills. 


e The fifth lesson would then be a VEIP-only session, and so on. 


Figure 1 gives an overview of how each element was intended to contribute to the development of 
pupils’ reading skills. 
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Figure 1. Vocabulary Enrichment Full Programme overview 
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Deliverers select material from these sessions according to 
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2.4 Outcomes 


The Vocabulary Enrichment Full Programme aimed to improve literacy, and, in particular, reading 
comprehension. The main component of the intervention, VEIP, was designed primarily to improve the 
speaking and listening skills of children with speech, language and communication difficulties. The 
theory of change behind the use of VEIP in this trial was that it would improve reading comprehension 
through: extending vocabulary; developing understanding of words in context (for example, how 
sentence construction affects meaning); and developing understanding of non-literal language. 
Sounds-Write (phonics) would provide further strategies for learning and recognising words, and 
Literacy Plus would provide the reading and writing context. 


Reading ability was chosen as the primary outcome, as improving reading ability was the main aim of 
the intervention. The test used was the paper version of the New Group Reading Test (NGRT; GL 
Assessment’), as it is a reliable test that has been standardised for the age group in question. The 
NGRT has two parallel forms: form A was used at baseline and form B at follow-up. The NGRT has 
two subscales, sentence completion and passage comprehension, which can be combined into a 


'http://www.gl-assessment.co.uk/products/new-group-reading-test 
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composite reading score. The composite reading score was used as the primary outcome. The two 
subscales were used as secondary outcomes. Given evidence from the process evaluation that 
teachers in three schools appeared to use VEIP almost exclusively, and the fact that VEIP’s primary 
aim is to improve speaking and listening skills, another secondary outcome testing these skills could 
have been useful. The evaluator did not foresee that the intervention would be adapted in this way, so 
another secondary outcome was not planned for. 


In order to minimise bias, Bolton LA was not involved in testing. School staff invigilated whilst pupils 
took the tests under exam conditions. Completed scripts were sent to GL Assessment for blind 
marking. Results were calculated using GL Assessment’s online system, and were accessed through 
GL Assessment’s online platform following testing. 


Whilst the developer was not involved in data collection, complete blinding at test administration 
cannot be assured as this was the responsibility of school staff, who were likely to have been aware of 
the intervention taking place and the allocation of pupils to the intervention group. Bolton LA was 
asked to ensure that different teachers administered the test than those that delivered the intervention. 
It was made clear to the schools that this was a requirement and there was some evidence that it 
occurred in the form of feedback from schools to Bolton LA. No formal checks on independent 
administration were carried out. 


2.5 Sample size 


Figure 2. Power curve 
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Power 


S = Number of participating schools 
P = Number of participating pupils per intervention group per school 


Randomisation was conducted at the pupil level and variation in baseline scores was controlled for in 
the final analysis. Intra-class correlation (rho) was therefore likely to have a minimal impact on the 
effective sample size so we have conservatively assumed a value of rho=0.02 for the purposes of our 
calculations. We also assumed a correlation of 0.75 between baseline and follow-up scores on the 
basis of previous work with reading tests. The power curve in Figure 2 illustrates that a sample size of 
528 will be sufficient to detect effect sizes at least of the order 0.20—0.25. This could be considered 
low-moderate, equivalent to around 3 months of progress; quite reasonable for targeted interventions 
providing support to small groups of pupils. 


During recruitment to the trial, it was decided to extend eligibility to Level 4b since a 4c cut-off would 
result in too few pupils being randomised (this was the EEF’s preferred option, rather than including 
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schools from another local authority area in the trial). This resulted in a total of 649 pupils being 
randomised, in excess of protocol sample size calculations after a projected 15% attrition. 


2.6 Minimum detectable effect size (MDES) 


Once all the data from the trial was available, the assumed parameters from the above calculations 
were compared to the actual parameters and included in a calculation of MDES. 


Randomisation was carried out at the pupil level, thus cancelling out the effect of clustering when 
estimating internally valid uncertainty around the effect. Rho can hence be regarded as zero. The 
adjusted R-squared for the primary outcome model without the intervention term was 0.47, implying 
that a value of 0.69 would have been more appropriate for the correlation between baseline and 
follow-up scores. Using the actual number randomised yields an MDES of 0.16 at 80% power. 


2.7 Randomisation 


Bolton LA and the individual schools involved in the trial were responsible for pupil recruitment. 
Randomisations were carried out by a statistician at NFER using a full syntax audit trail within SPSS. 
Randomisation was stratified by timetable half within each school: simple randomisation of pupils into 
two groups of the same size was carried out within each timetable half. This was necessary to aid 
timetabling of the sessions within schools. 


Schools required the results of the randomisation prior to baseline testing for timetabling reasons, and 
so these were released directly to Bolton LA who managed all communication with schools. 


2.8 Analysis 


The primary outcome was reading ability as assessed by raw scores from the NGRT. Raw scores 
were used in preference to age-standardised scores due to potential ceiling or floor effects in the 
latter. Subgroup analysis on the primary outcome was carried out on the following groups only: pre- 
test score, National Curriculum level and whether or not a pupil was eligible* for free school meals 
(FSM). The secondary outcomes were the two NGRT subscales: sentence completion and passage 
comprehension. All outcomes and subgroup analyses were pre-specified at the protocol stage. 


The definitive analysis was ‘intention to treat’, reflecting the reality of how interventions are delivered in 
practice. It was necessary to take school context into account in the analysis due to the restricted 
nature of the randomisation (Kahan and Morris, 2012). Eleven dummy variables were included in the 
regression model to represent school; one school was the default category. The definitive primary 
outcome analysis regressed post-test raw score on pre-test score, school, randomised group, sex, 
FSM and age in months. Subgroup analysis was carried out using a separate regression model on 
FSM pupils and by exploring the interaction between randomised group and pre-test score. Secondary 
outcomes were analysed using raw scores in the relevant domains in place of overall reading scores. 


The main analysis was followed by an ‘on-treatment analysis where data from teacher logs was used 
to determine the extent of each pupil’s involvement with the intervention. For each timetable half in 
each school, the total delivery time was used as a measure of dosage. Furthermore, group moves and 
withdrawals were also taken into account. The dosage measure was then used in place of the 
dichotomous group allocation. Other than one documented group move from control to intervention, 
control pupils were allocated a dosage of zero. This analysis allows for an estimate of ‘pure 


? The full definitions of the two states that this variable describes are: 1) eligible and registered/claiming; 2) not eligible or eligible 
and not registered/claiming. 
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intervention effect’ (net of any fidelity issues, contamination or non-completion). However, note that 
this analysis may be biased due to self-selection to differing levels of exposure.° 


2.9 Process evaluation methodology 


The process evaluation encompassed the entire duration of the intervention from the start-up meeting 
in December 2012 to completion of the intervention in February 2014. The evaluator collected 
information from a review of the training materials, observations of the training sessions, observations 
of intervention sessions in situ, telephone interviews with the teachers, and a review of the qualitative 
parts of the ‘teacher logs’. Further information was also gathered in informal discussions with the 
project coordinator at Bolton LA. These methods were chosen to ensure the evaluation team covered 
all the different elements of the intervention. 


Detailed schedules for the training observations, session observations and telephone interviews were 
developed to ensure that data collection was consistent and comprehensive. As more than one 
researcher was involved in observing training and intervention sessions and undertaking interviews, 
team meetings were held to share information and plan next steps. All researchers have contributed to 
the report writing to ensure full coverage of the information gathered. 


Session observations and teacher logs: Two sessions (English lessons involving the intervention 
group) were observed in November 2013. These sessions were chosen by randomly selecting from 
the participating schools and negotiating the visit date and time with the schools. Staff teaching these 
sessions also provided informal feedback during the researcher's visit. Teachers used a ‘teacher log’ 
pro forma to record an outline of each intervention session (date, duration, content) throughout the 
trial, and to note any deviation from their intervention plan in terms of content or, for example, the 
session being covered by someone other than the usual teacher. The data on the teacher logs was 
used to calculate the intervention dosage, i.e. the amount of time that each class was exposed to the 
intervention. Information from the teacher logs further informs the assessment of intervention 
implementation and fidelity. 


Telephone interviews: These were conducted in March 2014, i.e. at the end of the intervention 
period. Evaluation staff interviewed five teachers representing four schools (both teachers at one 
school wanted to be interviewed). They were selected at random from the participating schools, 
excluding schools where the evaluator had already observed intervention sessions. Interviews took 
approximately 40 minutes to one hour to complete, and researchers produced detailed write-ups after 
the interview. The interview topics included perceptions of the training, delivering the lessons, the 
resources required, perceived outcomes, issues of cross-contamination, and other issues such as 
perceptions of scalability. 


Observations of training sessions were conducted in June and July 2013. The evaluator observed 
two of the four days of Sounds-Write training and the two days of VEIP/Literacy Plus training. A further 
planning day for VEIP (in September) was not observed. 


° For example, pupil motivation may be positively related to both levels of exposure to the intervention (through better 
attendance) and the amount of progress made between baseline and follow-up testing. 
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3. Impact evaluation 


3.1 Timeline 


Table 1: Timeline 


January-February 2013 Recruitment and consent from schools (recruitment event held 25 
February) 

June-—September 2013 Random allocation of pupils on a school-by-school basis 

August 2013 Opt-out parental consent obtained 

September 2013 Pre-testing (9-13 September) 

September 2013 — Delivery of intervention programmes 

February 2014 

February 2014 Post-testing (10-14 February) 

March 2014 Control group pupils begin to experience intervention 


Randomisation had to occur before parental consent was obtained and before baseline testing for 
timetabling reasons (see Eligibility section). 


3.2 Participants 


All schools involved in this project were in Bolton LA area. Further details are provided in Tables 2 to 
5. All 17 Bolton secondary schools were approached by Bolton LA and the 12 agreeing to take part 
were required to sign a memorandum of understanding (see Appendix B). Recruitment of schools 
occurred during January and February 2013. The acquisition of parental consent occurred in August 
2013 by sending letters to home addresses which contained no reference to group allocation (see 
Appendix C). A timeline including recruitment, testing and intervention implementation can be found in 
Table 1. 


Of the 2150 pupils screened for eligibility, 649 pupils were deemed eligible for the study on the basis 
of predicted Key Stage 2 results and were randomised to the intervention or control groups. 


Table 2: Ofsted ratings for schools 


Outstanding { 
Good 7 
Requires improvement 4 
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Table 3: School type 


Comprehensive 11-16 8 
Comprehensive 11-18 2 
Academy* 2 


Table 4: Percentage of pupils eligible for FSM at school level 


Highest quintile 
Second highest quintile 
Middle quintile 

Second lowest quintile 
Lowest quintile 


O/}NM| 0) o|/— 


Table 5: School attainment 


Highest quintile 
Second highest quintile 
Middle quintile 

Second lowest quintile 
Lowest quintile 


Om) oO}—/| 4) 4 


All but one of the schools involved were urban secondary schools. Seven of the schools had a Good 
Ofsted rating, four were rated as Requires improvement and one as Outstanding. All twelve schools 
were mixed sex. Within the twelve schools there were slightly more falling in the higher FSM eligibility 
groups and low attainment groups. 


“ One of these two schools became an academy during the course of the trial and was previously an 
11-16 comprehensive. 


° Total GCSE points score averaged for school then weighted by number of pupils in school to 
establish quintiles. This was used in preference to percentage of pupils with 5 or more A*—C grades as 
it gives a better overall picture of attainment. 
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Figure 3: Participant flow diagram 


Assessed for eligibility (n= 2150; 
1990 in 11 schools were screened 
by Bolton LA, the number screened 
by the 12th school was estimated 
from the NFER Register of Schools) 


Excluded (n=1501; 
ineligible for trial) 


Randomised (n=649) 


Allocated to intervention (n=325). After Allocated to control (n=324). After consent 
consent and exclusion (n=312; see note 1) and exclusion (n=314; see note 1) 


Sat baseline test (n=292) Anton | Sat baseline test (n=295) 


Did not receive allocated intervention and Moved into intervention group and 
retained in study (n=24; see note 2) retained in study (n=1) 


Excluded (n=9; see note 3) Lost to follow-up (n=11) 
Lost to follow-up (n=10) Sat follow-test (n=303) 


Sat follow-up test (n=293) 


Final analysis (n=282) | Final analysis (n=288) 


Total number of randomised pupils not 


Total number of randomised pupils not | included in final analysis (n=36) 


included in final analysis (n=43) 
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Notes to Figure 3 


1. Between randomisation and pre-testing, some pupils were excluded from the trial, for the following 
reasons: their parents had opted out, or their attained Key Stage 2 results were too high, or they had 
left the school, or the parental consent letter was not delivered. All exclusions at this stage were made 
by individuals who did not know whether pupils were allocated to intervention or control. 


2. Of the 24 pupils that did not receive the allocated intervention but were retained in the study, 22 
were from a school that could not deliver the intervention to half of its allocated intervention pupils for 
timetabling reasons. 


3. Reasons for the 9 intervention group exclusions that occurred after baseline testing ranged from 
medical to, more commonly, teachers exercising professional judgement about the unsuitability of the 
intervention for certain pupils. These pupils ceased to receive the intervention and did not sit follow-up 
tests. 


4. For all these examples of attrition, the evaluator advised the schools through Bolton LA to respect 
the original groupings and to retain the pupils for testing. For the 22 pupils whose school could not 
deliver the intervention, 15 were included in the final intention-to-treat analysis. 


3.3 Pupil characteristics of analysed groups 


Whilst we expect no systematic bias to have arisen from randomisation, bias may have occurred due 
to attrition. Chi-squared tests on the three background factors presented in this section revealed no 
significant differences between groups for the data after attrition. 


Table 6: National Curriculum level in reading at baseline (y*= 2.3, df=2, p=0.31) 


3 70 25 69 24 
4 184 65 185 64 
5 12 4 21 7 
Missing 16 6 13 5 
Total 282 100 288 100 


Table 6 shows that vast majority of pupils satisfied the eligibility criteria. It should be noted that 
schools were using predicted Key Stage 2 results to determine eligibility, so some improvement may 
have occurred since then. 


Table 7: FSM eligibility (x= 0.25, df=1, p=0.61) 


Yes 81 29 77 27 
No 201 71 210 73 
Missing 0 0 1 0 

Total 282 100 288 100 
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Table 8: Sex (x= 0.66, df=1, p=0.41) 


Male 171 61 165 57 
Female 111 39 123 43 
Missing 0 0 0 0 

Total 282 100 288 100 


Table 9: Pre-test raw scores 


Mean 22.7 22.9 
Standard 6.3 6.9 
deviation 

n 282 288 


Further to pupil background measures, it was also important to test whether significant imbalance at 
pre-test had ensued as a result of attrition. The baseline effect size was -0.03 (-0.20, 0.13) and was 
not significant (p=0.705). 


3.4 Outcomes and analysis 


Table 10: Effect size 


Primary Reading score 0.06 -0.06 0.18 0.346 282 288 
(NGRT) 

Primary Reading score 0.01 -0.23 0.25 0.948 = 81 77 

(FSM) (NGRT) 

Secondary Sentence 0.02 -0.12 0.16 0.804 282 288 
completion 
score 

Secondary Passage 0.07 -0.05 0.19 0.281 282 288 
comprehension 
score 


An ANOVA of post-test reading score by randomised group showed no significant impact of the 
intervention (F=0.27, p=0.60, n=596). Table 11 contains descriptive outcome statistics for analysed 
groups. 


Table 11: Post-test raw scores 


Mean 23.0 22.7 
Standard 6.8 7.3 
deviation 

n 282 288 
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All outcomes analysed were pre-specified in the protocol. All subgroup analyses were pre-specified in 
the protocol aside from the use of FSM as a proxy for pupil premium; a separate FSM analysis is a 
requirement of all EEF evaluations. Background data on pupils was obtained both from schools 
through the standard GL Assessment data form and from the National Pupil Database (NPD). Where 
the same variable was obtained, the latter source was used in preference unless it contained fewer 
cases. The main primary outcome analysis consisted of a regression model containing the following 
variables: pre-test score, intervention group and school. FSM, age in months and sex were included in 
an initial run of the model but none was significant so all were excluded. The use of a linear regression 
model represents a slight deviation from the protocol where it was specified that a multi-level model 
would be used. Because randomisation was restricted by school, it was necessary to account for 
school in the model but this was done using dummy variables rather than a multi-level model. This 
method was used in preference since a multi-level model would not have estimated variance 
adequately with only twelve schools. As this was a pupil-randomised trial, clustering for the 
intervention effect is not an issue in terms of internal validity. Model results are presented in Appendix 
A. 


The protocol specified both an analysis by national curriculum level and pre-test score. As pre-test 
score is the best resolution of reading ability measure available, this is a more sensitive analysis. The 
interaction between intervention and pre-test score was tested in a model equivalent to that used in 
the primary analysis but including the interaction term. The interaction was not significant (p=0.47). 
The interaction between intervention and level was tested using a model equivalent to that used in the 
primary analysis but including a dummy variable for Level 3 (the default was Level 4/5) and the 
interaction term. The interaction was not significant (p=0.91). 


All the above analysis was ‘intention to treat’. The ‘on-treatment’ analysis used a measure of 
intervention experienced by each pupil in terms of total delivery time as declared by teachers in their 
logs (see ‘Analysis’ section). The dosage was highly variable, ranging from 33 to 85 hours with a 
median dosage of 53 hours. The ‘on-treatment’ analysis, taking into account the extent of delivery and 
those that switched groups, revealed no significant effect of dosage (p=0.38). 


3.5 Cost 


The project had core and variable costs, as described in the sections below. The minimum (core) cost 
per school to train and equip four teachers to deliver the intervention to 100 pupils would be around 
£7,470 (£75 per pupil). The maximum cost (core plus variable costs) per school could be in the region 
of £10,990 (£110 per pupil). 


Core costs 


The schools in the trial did not report incurring any costs, as the funding that Bolton LA received to 
conduct the trial covered their costs. The project budget covered the cost of staff cover for seven days’ 
training per teacher (i.e. training for all intervention components); this cost £1,400 per teacher. 
Sounds-Write training (including resource materials) costs approximately £380 excluding VAT per 
delegate (the VAT should be reclaimable by a school). The VEIP folder costs £87.50. 


Assuming a school trained four of its existing teachers to deliver the intervention, the core cost 
(Sounds-Write training, cover costs during seven days’ training, and the provision of one VEIP folder 
per teacher) would be £7,470. 


Variable costs 


VEIP and bridging activity training (preparation and delivery) had a budget of £5,200. Literacy Plus 
training preparation and delivery had a budget of £7,000. These were bespoke training activities so 
the cost of training to further schools is difficult to estimate. If the costs had been split equally between 
the 40 delegates, the per delegate cost for VEIP training would be £130; for Literacy Plus, £175. 
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In the trial, the schools accessed support from an educational psychologist employed by the LA 
(covered by the project budget); if a school wanted to procure this support for one or two hours once a 
month for six months, at approximately £60-80 per hour, this would cost around £800 in total. 


Each school also received £1,500 for photocopying/resources. 


Assuming that a school wanting to train four teachers would be responsible for these variable costs 
(VEIP and Literacy Plus training, support from one _ educational psychologist, and 
photocopying/resources), the cost to the school in addition to core costs would be in the region of 
£3,520. 


The project also funded the creation/teaching of two additional pupil groups (the intervention groups) 
per school, costing £10,250 per school for most of the trial schools (this included the £1,500 for 
photocopying/resources). Schools outside the trial would not necessarily need to create additional 
pupil groups, depending on how they implemented the intervention (e.g. across the year group or fora 
specified group of pupils needing to be taught separately). Schools intending to deliver the intervention 
would also need to consider providing extra non-contact time for teachers to prepare the scheme of 
work for the lessons; some teachers reported that the time required went beyond their normal planning 
and preparation time allocation. 


Apart from the programme folders/lesson plans, resources required included stationery, mini 
whiteboards (if used) and dictionaries, all of which would usually be available in schools. Teachers 
commented that delivering VEIP activities involved a lot of photocopying, although there was budget 
allowed for photocopying/resources in the project funding. 
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4. Process evaluation 


The project was initiated by Bolton LA. The key components of the intervention and the trial design 
were agreed after liaison between the LA and some of the schools that had expressed an interest in 
participating. Schools agreed to participate in the intervention and evaluation before training took 
place (detailed below). Recruited teachers were offered further support during the intervention period 
in the form of regular visits from an educational psychologist. Teachers met each half term to discuss 
sessions and share reflections. They also had access to a virtual learning area (website) to share 
suggestions for activities with all the other teachers taking part in the trial. 


4.1 Training observations 


Sounds-Write training observation (training session 1, June 2013) 


The standard Sounds-Write training is a four-day course; the final two days were observed. At the end 
of the four days of training, delegates needed to complete and pass an assessment in order to pass 
the training. 


The training was led by John Walker, one of the creators of Sounds-Write, and another trainer from 
the company. A learning consultant, who is based at one of the trial schools and was involved in 
creating the Vocabulary Enrichment Full Programme, also contributed to the training. The training was 
held in a classroom at the consultant’s school. There were thirty delegates, mostly teachers who were 
going to deliver the intervention. The limited space available in the room and external noise was not 
ideal for the training. 


All delegates received a folder containing Sounds-Write resources and detailed plans for each session 
(not available separately from the training). The programme design expects teachers to adhere to 
these plans rather than using them as suggestions for their own planning. However, this approach was 
not suited to, and therefore not generally adopted in, this research project, where elements of different 
interventions were meant to be combined into one session. 


For the two days observed, delegates were guided through the programme folder and worked in pairs 
to role-play the sessions. The trainer reported that the first two training days had been similar, with a 
general introduction to Sounds-Write at the beginning of the training. Engagement with the training 
appeared mostly good, and most delegates were enthusiastic about using the techniques. However, 
others felt that too much of the training was spent on phonics programme basics before moving on to 
polysyllabic words (the sort of words that will be encountered regularly at secondary level). Therefore, 
the applicability of the programme to secondary school children did not become apparent until the later 
stages of training. Some delegates reported that the training was too long and intense, given that 
VEIP, rather than Sounds-Write, was the primary method in the intervention. There was no option to 
offer more concise Sounds-Write training as it is delivered as a standard four- or five-day course. 


Vocabulary Enrichment/Literacy Plus (training sessions 2 and 3, July and September 
2013) 


The first of two days of training in July focused on the Vocabulary Enrichment Intervention Programme 
(VEIP). This session was led by a speech and language therapist (SLT) and an educational 
psychologist (both from Bolton LA), with support in the activity session from a further four educational 
psychologists and two SLTs. In addition to preparing staff to deliver the intervention, the training aims 
to improve teachers’ knowledge and awareness of speech, language and communication difficulties. 
There were forty delegates, mostly teachers who were going to deliver the intervention. The teachers 
appeared engaged throughout the morning and reported that the training had been well-delivered and 
useful. In the afternoon, teachers were asked to plan a VEIP session with support from the 
educational psychologists and then deliver it to the group. However, no teachers volunteered to deliver 
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the planned session as they felt they had not had enough preparation time. Instead, they either 
discussed the programme in their groups or continued planning. 


The second training day in July was led by a learning consultant based at one of the participating 
schools, with input from the Bolton local authority/educational psychology service, and the project 
lead. Again, there were forty delegates, mostly delivery teachers. The trainers gave a brief overview of 
Literacy Plus, which delegates reported being comfortable with as it is similar to their usual English 
teaching strategies. The primary focus of the day was on how the three schemes would be brought 
together to create one intervention. Delegates were given examples of how a lesson could combine 
elements of VEIP, Sounds-Write and/or Literacy Plus, plus a bridging/plenary activity to link learning to 
the wider curriculum or personal reading, and guidance as to which code words in VEIP could be 
tackled with Sounds-Write approaches. Delegates had time to start planning sessions. The evaluator 
also briefed delegates on the practical aspects of the trial and evaluation. As with the previous day, 
the delegates’ initial high engagement diminished as the day progressed. During breaks, delegates 
commented that the breaks were too long, and that they wanted to finish training and start planning. 


An additional training day for teachers in September focused on planning VEIP-based lessons. This 
was not observed. 


Intervention delivery 


The two lesson observations (one at each of two schools), post-intervention telephone interviews with 
five teachers (at four schools) and information from the teacher logs (completed by all teachers) inform 
this section. 


As discussed in the Methodology section, delivery of the intervention was not strictly prescribed and 
teachers were encouraged to adapt the intervention (for instance, the combination of the three core 
elements and use of stimulus materials) to their own needs, as teaching professionals might do when 
creating any scheme of work. If pupils did not respond well to a particular activity or stimulus, or were 
slow to complete an activity, the teacher moved on to the next activity and/or adjusted their lesson 
plans as per their usual teaching practice. 


Teachers were directed to use VEIP as the main content of the lessons, and to feed in Sounds-Write 
activities and Literacy Plus material as they felt appropriate. In practice, teachers all used VEIP 
throughout the intervention period. Due to the different ways teachers selected from the VEIP material 
and the pace at which pupils progressed, some classes worked through up to the final (24th) session 
of VEIP by the end of the intervention period, while others reached session 14 or 15. Teachers’ use of 
Sounds-Write and Literacy Plus was much more variable. The teacher logs indicate that several 
teachers (eight schools) did not explicitly include Literacy Plus at all, and a few teachers used it in one 
or two dedicated lessons. One teacher reported there was no need to use Literacy Plus (preferring to 
use the normal class set texts if additional reading material was needed), and another indicated that 
the Literacy Plus material soon became very ‘dry’ and unengaging, so its use was discontinued and 
normal class texts were used instead. Several teachers explicitly used Sounds-Write throughout 
several sessions, a few teachers had dedicated two or more lessons to it, and others had used it 
minimally. One interviewee said Sounds-Write informed all VEIP teaching. However, the other 
interviewed teachers had generally not used much Sounds-Write and or Literacy Plus. Some of these 
teachers commented that planning was very time-consuming so it made sense to focus on VEIP as it 
was meant to be the main component of the intervention, and to use texts that they were already 
familiar with and/or preferred to Literacy Plus. One also commented that there was not sufficient time 
in the lessons to use Sounds-Write in the focused, specialist way that the programme designers 
intended, so they had ‘only paid lip service to phonics’. Teachers at three schools used VEIP (almost) 
exclusively, and it does not appear that they used alternative texts to ‘replace’ Literacy Plus materials. 
One school used a further programme, the Narrative Intervention Programme, in addition to VEIP and 
Sounds-Write, but did not use Literacy Plus at all (the Narrative Intervention Programme, NIP, was 
devised by the developer of VEIP; schools in the trial each received one NIP folder during the 
training). Seasonal events and scheduled themed days naturally influenced the activities used in 
certain lessons (e.g. Christmas, anti-bullying week). 
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On occasion timetabled lessons were not used to deliver the intervention as anticipated, but to do 
standalone assessment tasks such as producing a piece of creative writing. Interviewed teachers 
explained that there was a need to formally assess pupil progress at an interim point during the 
intervention period, because school policy required that all Year 7 pupils be assessed. Some teachers 
included aspects of their usual scheme of work while using VEIP approaches to learning vocabulary; 
at one school, one third of the time was spent on the normal scheme of work in this way. Three out of 
four schools interviewed made these adaptations so that the intervention group would not ‘miss out’ on 
content in the usual scheme of work. These adaptations included stimulus texts and pupil 
assessments. One teacher commented that the intervention was fairly similar to their normal approach 
to teaching Year 7 pupils struggling with literacy. 


The sessions observed lasted 50 minutes and one hour respectively. Both sessions took place in a 
school classroom and followed the school’s usual literacy lesson practices in terms of format (i.e. 
starter, main activities, plenary). The first session included eleven pupils and the second included 
seven pupils, similar to the class sizes of other lower ability groups in the schools. The teachers 
followed their normal classroom practices in terms of behaviour management and use of ICT 
(interactive whiteboard and laptop). 


The first observed lesson focused on session 8 of the VEIP programme, in particular, homonyms. 
Once the learning objective had been introduced and discussed, the session included completing 
extended word maps, drawing pictures of homonym pairs and discussing definitions and examples of 
homonyms (and similarities/differences of objects in general). Pupils appeared engaged and coped 
well with the activities. In this observation, Sounds-Write and Literacy Plus were not used explicitly 
(but the teacher reported that s/he did use these in other lessons). 


The second observed lesson combined elements from the three different programmes (VEIP, Sounds- 
Write and Literacy Plus). The teacher felt that the Sounds-Write and VEIP programmes fitted well 
together and enabled cross-curricular links to be made (for instance, there are science-related VEIP 
activities). The observed lesson focused on descriptive writing and on revising some concepts that 
had been taught in previous lessons, including the parts of speech, conjunctions and different 
sentence types. Pupils were encouraged to read the learning objectives and stimulus material aloud 
and were very engaged in the activities. 


Both observed teachers believed that the children found the lessons engaging and that the work was 
at an appropriate level of difficulty. They criticised the amount of lesson planning involved. For 
instance, at the first school, the teacher stated that planning for the intervention took more time than 
planning for ‘normal literacy lessons at the school, as the activities and their sequencing are different. 
However, both teachers found the resources (programme folder) to be very helpful in planning, and 
commented that planning would be more straightforward in a second year of delivering the 
intervention. 


Facilitators to delivering the intervention 
Enjoyment and engagement 


Teachers enjoyed delivering the intervention, and tended to comment that the pupils enjoyed the 
VEIP-focused lessons in particular because the activities were usually engaging and pitched at the 
right level. One teacher mentioned that Sounds-Write worked particularly well with the lowest-ability 
children as pupils were familiar with it from primary school. 


Suitability of intervention within the English curriculum 


Two teachers commented that VEIP had important similarities to their usual Year 7 scheme of work 
(for instance, learning about parts of speech). Most of the teachers who were interviewed commented 
that their schools were considering integrating VEIP into the scheme of work for Year 7 next year, and 
would probably focus on the first ten sessions of the VEIP which built the foundation vocabulary skills. 


Flexibility and differentiation 
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Teachers valued the following: the flexibility of the VEIP programme meant they could select from the 
materials; the diagnostic capabilities of the activities (for instance, monitoring spelling and 
pronunciation through Sounds-Write); and the scope for differentiation in activities and in the level of 
response pupils could give. 


Interest and support from other staff 


Teachers perceived that non-involved members of staff, including senior leadership, were positive 
about the intervention. Teachers felt that other staff would be interested in delivering the VEIP 
intervention, as long as it was during normal English teaching time (and not an additional or 
withdrawal activity) in order to embed skills in the appropriate context. 


Additionally, two teachers commented on the importance of having suitable dictionaries which are 
comprehensive enough to include all the words that children encounter in VEIP sessions. 


Barriers to delivering the intervention 
Timetable logistics 


One school was not prepared to deliver the intervention across both timetable halves so 22 pupils who 
had been randomised to receive the intervention were in effect moved to the control group, although 
they remained in the intervention group for the purposes of the intention-to-treat analysis. For schools 
that divide up the year group in a similar way, it may prove logistically difficult to arrange separate 
classes for the intervention group. 


Planning and preparation time 


Interviewed teachers tended to criticise the amount of planning and preparation involved, reporting 
that the VEIP programme folder is very ‘wordy’ and ‘content-heavy’. As VEIP was time-consuming, 
teachers tended to pay less or little attention to including Sounds-Write and Literacy Plus, preferring 
their own class texts. Each session of the VEIP programme includes several activities, which teachers 
needed to select from according to their pupils’ needs and time available. Teachers therefore needed 
to spend a considerable amount of time working through, distilling and translating the material to 
realistic lesson plans. Some teachers also criticised the fact that the VEIP content is not in electronic 
form, meaning time spent typing content up into worksheets. Teachers usually planned in pairs and 
did some of their planning in school holidays, but teachers at some schools reported having less 
dedicated non-contact time for planning than others, and felt that more planning time should be 
allocated. However, teachers also said that the programme folder contained very useful activities and 
guidance, and one commented that the workload associated with the intervention was similar to 
planning any new scheme of work. 


Unsuitability of science-related lesson plans 


All five of the interviewed teachers commented that they disliked the science-related component of 
VEIP and felt it was unsuitable for participating pupils. They reported that it took too long for the 
children to comprehend the material and to acquire the knowledge and vocabulary involved. Although 
they appreciated the value of cross-curricular links and enriching vocabulary across subjects, they felt 
that there was too much crossover with the science curriculum and that it would be better if science 
teachers taught this vocabulary. They thought that the VEIP programme should be grounded solely in 
the English curriculum. 


Lack of English teaching specialism 


One teacher commented that, in her experience of the trial, teachers who did not have a background 
in teaching English found planning and delivery of the intervention more challenging than English 
specialists. This is likely to be true of many literacy-focused interventions, however. Interviewed 
teachers did not find the support from a visiting educational psychologist very helpful; most are 
experienced English teachers and some are also particularly experienced in literacy interventions so 
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did not need help with lesson planning or strategies, but this support may have benefited other non- 
specialists. 


Pupil engagement 


On the whole, interviewed teachers reported a good level of engagement and focus by pupils. In two 
schools, however, there was some evidence of disruptive behaviour during the first term of the 
intervention. In one of these schools, two pupils who were noticeably more able than their classmates 
started to become disengaged due to the lack of challenge. They were moved out of the intervention 


group. 
Relevance of Literacy Plus and Sounds-Write 


As detailed above, two interviewed teachers perceived Literacy Plus materials as unnecessary or 
unengaging and did not use the programme (or stopped using it early in the intervention period), and 
several others did not record any use of the materials in their teacher logs. Three of five interviewed 
teachers commented that the Sounds-Write material was aimed at younger pupils, so was largely 
irrelevant to their pupils. 


Lack of pupil assessment in delivery schedule 


Teachers felt that the lack of built-in pupil assessments or guidance on measuring pupil progress were 
significant omissions in the intervention. The volume of VEIP material they felt they needed to cover 
also put pressure on time for reflection and adaptation of future sessions. They perceived that interim 
assessments were important to delivering the intervention effectively, and said that it was essential 
that all Year 7 pupils did the usual assessment in English (a piece of creative or essay writing with an 
associated mark scheme, for example) in order to give feedback to parents and school senior leaders. 


4.2 Fidelity 


As discussed in detail above, there was not a prescribed way in which teachers should combine the 
three components (VEIP, Sounds-Write and Literacy Plus). The evidence shows that teachers were 
able to adapt the intervention, both when planning lessons in advance of the trial and, if necessary, 
during the trial to keep pupils engaged and working at an acceptable pace. Their primary motivation in 
doing so was to meet the needs of their pupils, but adaptations were also motivated by limited time for 
planning (for instance, most interviewed teachers indicated that there was limited time to incorporate 
Sounds-Write effectively), the desire to incorporate certain texts and activities from their usual 
schemes of work, and the need to include year-wide activities such as written assessments. Teachers 
all followed guidance from the project coordinator that VEIP material should be the main focus of the 
lessons, but did not all incorporate Sounds-Write and Literacy Plus regularly throughout the trial 
period. 


The number of children in a class and the organisation of the lessons were also not prescribed, and so 
also varied. For instance, some schools had three one-hour lessons a week; others had four 50- 
minute lessons. Over the course of the 19-week delivery period, the maximum duration of exposure to 
the intervention (dosage) would have been estimated at between 3,420 minutes (four 50-minute 
lessons per week) and 3,800 minutes (three 60-minute lessons per week). The range of actual 
dosages among intervention groups was much wider than this estimated range, assuming that the 
information on lesson duration/content on the teacher logs is accurate. The impact evaluation has not 
found a significant relationship between dosage and test score, however. Lessons were delivered 
consistently by the trained teacher, with very few exceptions where lessons were covered by other 
teachers or TAs (due to staff sickness, for example). Schools followed their normal practice in terms of 
the use of TAs in the class, so some classes had one or two TAs. 


As mentioned in the ‘Barriers to delivering the intervention’ section, at one school one of the two 
intervention groups did not receive the intervention. This is accounted for in the on-treatment analysis. 
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4.3 Outcomes 


Interviewed teachers were confident that the intervention resulted in benefits for pupils in terms of their 
reading and speaking abilities. They felt that VEIP activities especially had successfully engaged 
pupils and tended to link positive outcomes for pupils to VEIP; one said: ‘it has been incredibly 
powerful in accelerating progress. It has opened the door to understanding multiple texts.’ In 
particular, teachers mentioned that they perceived improvements in: 


e pupils’ interest in words 

e pupils’ recognition and recall of new/recently learned words 

e pupils’ understanding of phonics and pronunciation/spelling relationships 
e pupils’ understanding of word classes (e.g. antonyms, synonyms) 

e the variety and complexity of pupils’ vocabulary and sentence construction 


e pupils’ overall reading ability. 


Teachers did not report any specific negative impacts of the intervention on the pupils, although one 
teacher was concerned that, as their lessons focused almost exclusively on VEIP, the class might be 
missing out on developing other reading skills such as inference. 


4.4 Formative findings 


Interviewees were positive about the perceived outcomes and potential impact of the intervention. 
They generally felt that amendments to the training for, and content of, the intervention should be 
made to facilitate its wider implementation. 


Reconsider use of Literacy Plus and Sounds-Write 


In general, Literacy Plus and Sounds-Write were perceived by teachers to be less appropriate or less 
essential than VEIP. They suggested that the weakest-ability pupils would be most likely to benefit 
from Sounds-Write. Sounds-Write is aimed at a younger audience than Year 7, but unfortunately there 
is a lack of phonics programmes specifically designed for secondary school pupils. The inclusion of 
these programmes may reduce the acceptability of the intervention to other teachers, particularly if 
there are concerns that pupils may be ‘missing out’ on potentially more engaging and suitable 
activities from the usual scheme of work. 


Streamline the training 


The training could be more effectively delivered as skills-set training, encompassing the strategies 
used across the three component programmes and with an overview of how to diagnose pupils’ 
weaknesses and set objectives. As Sounds-Write has a prescribed training course, there is limited 
flexibility to design such as training programme if Sounds-Write is to be included. 


Revise VEIP resources 


The programme resources folder for VEIP (lesson plans and guidance notes) could be revised to be 
more concise and easier to navigate, with photocopiable pages that are ready to use in class. VEIP 
could also be improved by having clearer guidance and materials dedicated to formative pupil 
assessment. None of the interviewed teachers would use the science-related content in VEIP if they 
were to use the programme for future Year 7 cohorts, due to its difficulty for pupils and the overlap 
with the science curriculum, suggesting that schools should consider carefully whether it is appropriate 
for their school. 
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Allow sufficient time for planning and preparation 


Interviewees felt that other schools would find the intervention beneficial, but that the logistics and 
resources needed to create separate teaching groups for the intervention, and the need to allow 
teachers a substantial amount of non-contact time for lesson preparation, might prove to be barriers. A 
commitment to the programme by school senior leaders would be essential to ensure that this 
programme could be implemented successfully. 


4.5 Control group activity 


Teachers at three of the four schools represented in interviews described the control group lessons as 
the ‘usual Year 7 scheme of work’, as specified in the trial protocol. These teachers said that they did 
not share resources or strategies with other staff in the participating schools. One of these schools 
decided, after delivery had begun, to use the same class text with both intervention and control 
groups, as children were disconcerted that they were initially using different texts. This enabled 
children to discuss what they were reading across intervention/control groups and they may have also 
discussed the learning activities based on the text, although it seems unlikely that this knowledge 
would have benefited the control group. 


Session observations and interviews indicated that there was some contamination of the control group 
in one school, where non-delivery staff had used elements of VEIP and Sounds-Write in the Year 7 
control group. For instance, teachers had used VEIP activities as starter activities with the control 
group. The teachers at that school explained that control groups’ lessons were different to intervention 
groups’ lessons as they would use different levels of texts; or where a control group might have a 
VEIP activity as a starter activity before moving on to the usual scheme of work, the intervention group 
would focus on the activity for most of a lesson. 
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5. Conclusion 


5.1 Limitations 


In one school, there was evidence of contamination of delivery to the control group, whose experience 
was not therefore ‘business as usual’. The teachers had presented some of the programme material at 
a departmental meeting, and subsequently non-delivery teachers had used some activities from VEIP 
and Sounds-Write as starter activities with control pupils. Interviews and discussions with teachers 
following session observations (involving six schools overall) did not suggest that contamination of the 
control group by staff was more widespread than this, however. 


It is also possible that there might have been contamination of the control group as a result of 
interaction between intervention and control pupils. Intervention pupils might have discussed the texts 
and activities experienced in their lessons with control pupils. This kind of contamination risk is 
apparent in most pupil-randomised trials and the type of information shared between pupils is unlikely 
to represent a further risk to the validity of the result. 


Teachers who delivered the intervention were aware that they should not be involved in administering 
the tests, but as the evaluators did not observe the testing we cannot be sure that tests were 
administered blind. 


In the trial, new teaching groups were created in order to deliver the intervention during timetabled 
English lessons. Control pupils may have benefited from being in classes of reduced size. 


Due to school timetabling requirements, randomisation had to occur in advance both of opt-out 
parental consent and baseline testing. Despite assurances from Bolton LA that no child was excluded 
from the study by someone with knowledge of their group membership, randomised lists were with 
schools and this remains a possibility. There was no evidence of biased attrition, however, and the 
number of exclusions at this stage was small. 


Overall attrition between randomisation and analysis was 12%. This is satisfactory, particularly given 
that parental consent was obtained after randomisation. This level of attrition, in combination with the 
balance seen in baseline characteristics of analysed groups, suggests that attrition is unlikely to be a 
problem for this trial. 


The NGRT was considered an appropriate test of the package of literacy skills developed through the 
Vocabulary Enrichment Full Programme, and teachers expected and observed that improved reading 
comprehension was an outcome of the intervention. Given that the main component of the intervention 
(VEIP) was vocabulary enrichment with grammatical knowledge, however, and as three schools 
seemed to focus (almost) exclusively on VEIP, an additional test that focused on vocabulary skills 
might have been more suitable. Vocabulary and grammatical knowledge are among the skills tested 
by the NGRT. VEIP does not specifically aim to develop other skills tested by the NGRT, including 
inference, deduction, and recognising authorial intent. These skills are explicit objectives of the 
Literacy Plus programme, but some teachers more or less ignored Literacy Plus. These teachers 
should have used other material to ensure that other literacy skills were being addressed, and the 
evidence suggests that teachers usually did this, with the three exceptions mentioned above. In one 
school, teachers were concerned that pupils were not developing these skills because they were 
spending most of their time on VEIP. In cases such as these, the NGRT would have tested a wider 
skills set than pupils could be expected to have developed. 


This trial was run as an efficacy trial in Bolton secondary schools. As it was delivered by teachers in 
schools rather than by a developer with specific knowledge of the intervention, the result is more likely 
to be applicable to a real-world scenario. However, whilst internally valid, this trial has little external 
validity from a strict statistical perspective. The schools recruited into the trial were local to Bolton and 
therefore were not randomly selected for participation in the trial. Consequently the sample cannot be 
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said to be representative of any population of schools beyond Bolton. However, there is nothing to 
suggest that Bolton is atypical of urban LAs. 


5.2 Interpretation 


The main result of the trial was that VEFP had no effect on reading ability, as compared to schools’ 
normal Year 7 English curriculum. 


Although this trial should be regarded as an efficacy trial it was the first time that the intervention was 
delivered as a package (i.e. VEFP). Furthermore, teachers adapted the intervention in different ways. 
It would have been beneficial to have first had a more extensive pilot phase to determine what was 
likely to be an appropriate and feasible combination of components to be used in the Full Programme 
intervention. This trial demonstrated that, when delivering together, teachers may have prioritised 
VEIP at the expense of the other elements. 


The adaptations to the intervention made by teachers tended to make lessons more similar to their 
usual scheme of work than if they had used the intervention resources/strategies exclusively. This 
may have reduced the potential for impact of the intervention. 


The NGRT test seems to be an appropriate test of the skills that VEFP was designed to develop. It is 
not, however, the most suitable test for the vocabulary skills that the intervention emphasised in 
practice (and some teachers focused on almost exclusively) as it tests a wider skills set. A vocabulary- 
based test may have indicated a greater intervention impact. 


The main component of VEFP (the VEIP) had been previously evaluated using an RCT. VEIP’s 
developer concluded that it was effective in improving vocabulary test scores of children with speech, 
language and communication needs, but it has not been possible to trace a full report of this RCT. The 
remaining two components, Sounds-Write and Literacy Plus, had not been tested against a 
comparison group so any reported improvements could have been due to changes that would have 
happened anyway, without the intervention. As all pupils at or below Level 4b were eligible for the 
VEFP rather than only those children with speech and language difficulties and as this trial measured 
reading ability rather than vocabulary specifically, it is not surprising that this trial should yield different 
results. 


5.3 Future research 


There is a lack of published, independent evidence on the effectiveness of each individual element of 
the intervention. Teachers’ feedback on the Sounds-Write programme suggests that it is not 
particularly suitable for secondary school aged pupils, but there does not seem to be a more 
appropriate phonics programme available for this age group. After obtaining robust evidence of the 
effectiveness or otherwise of each individual element, it would be informative to conduct a pilot project 
to evaluate the chosen elements in a more strictly specified combination. Originally this trial was to 
have three arms (Full Programme; Full Programme without VEIP; Control) but potential participant 
schools felt that this would be too difficult to implement within their existing timetabling arrangements. 
The design and resourcing of a further pilot would need to be carefully considered to encourage 
schools’ participation. 


The primary outcome of this evaluation was reading ability, as measured by the New Group Reading 
Test. VEIP is designed to improve pupils’ vocabulary, so further research could examine the impact on 
vocabulary as the primary outcome. 
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Appendix |: Model results 


Results of main effect model: 


Coefficients* 


Standardised 
aca | Coefficients Coefficients 


(Constant) 
Basescore 
intervention 
school1 
school2 
school3 
school4 
school5 
school6 
school7 


school8 


school9 


school10 


school11 


a. Dependent Variable: Followscore 


Neither female, FSM nor age in months was significant so these were excluded from the model. 


Education Endowment Foundation 31 


y Vo) eY-ale|[e{--) 


Results of FSM model 


Coefficients* 


Standardised 
aaa Coefficients Coefficients 


(Constant) 
Basescore 
intervention 
school1 
school2 
school3 
school4 
school5 
school6 
school7 
school8 
school9 
school10 


school11 


a. Dependent Variable: Followscore 
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Appendix Il: Contract for schools 


Agreement to participate in the Bolton EEF Literacy Transition Project 


School: 
Aims of the Evaluation 


The Bolton EEF Literacy project aims to determine whether the use of the Vocabulary Enrichment Intervention 
programme (VEIP) coupled with the teaching of phonics (Sounds-Write) and elements of Literacy Plus is effective 
in accelerating the literacy skills of Year 7 pupils. Two English teachers within each school will be trained to 
deliver the programme so that, if successful, it can be embedded sustainably in the future work of the school. 


The results of the research will contribute to our understanding of what works in raising the pupil’s attainment and 
will be widely disseminated to schools in England. Ultimately we hope that the evaluation will equip school staff 
with additional knowledge and skills with which to better support children with reading. 


The Project 


The Bolton EEF Literacy Project consists of an intervention and control group(s) in each school. Participating 
pupils will be working at Level 3 or 4c at the end of Key Stage 2 in reading. The intervention will be delivered 
during timetabled English lessons between September 2013 and February 2014. The intervention group in each 
school will receive a combination of VEIP, sounds write and elements of Literacy Plus, whilst the control group will 
remain in their normal school routine and so access the English curriculum as usual. The impact of the 
intervention will be measured by NFER who will assess the pupils using the GL Assessment New Group Reading 
Test (NGRT) both pre and post intervention. 


Structure of the Evaluation 


Participating High schools will be informed of all eligible pupils due to transfer to their school in September 2013. 
The schools will then place these pupils into the bands/populations within their school. This information will then 
be sent to NFER who will randomly allocate eligible pupils within each band/population into either a ‘control’ or 
‘experimental’ group. All eligible pupils will then complete the baseline tests using NGRT during the first 2 weeks 
of September 2013. Post intervention tests using the NGRT will be completed during February 2014. Across all 
schools involved, researchers from the evaluation team will observe two intervention sessions in each school and 
also a training session. Teacher logs and telephone interviews will provide further information to contribute to the 
process evaluation. Following the completion of the project in February 2013 teachers will be free to deliver the 
VEIP/sounds-write intervention to pupils in the control group. 


The evaluation is being conducted by Ben Styles and Rebecca Clarkson from NFER. Pupils who are eligible and 
whose parents agree® for them to take part will be placed into a population/band by school and then randomly 
allocated to either the intervention group or control group. Random allocation is essential to the evaluation as it is 
the only way that we can say for sure what the effect of the intervention is on children’s attainment. It is important 
that schools understand and consent to this process. Random allocation will be carried out by NFER so named 
pupil data will be collected by the project team and shared with NFER via a secure portal. 


The pupils in the intervention group will receive a combination of VEIP, sounds write, literacy plus and top-up their 
normal English curriculum, where appropriate, during their timetabled English lessons. The pupils in the control 
receive their usual school experiences. 


The New Group Reading Test (NGRT) 


°Via a passive consent letter. 


Education Endowment Foundation 33 


7 No) oY-ale|[e-t-) 


As part of the evaluation it is required that all the pupils selected take the paper version of the New Group 
Reading Test before the intervention commences and after the intervention has finished. For both of these 
sessions the following will apply: 


The school will be given a ‘testing window’ and it is essential that ALL pupils sit the test within this time period. 
Process 


1. Sue Cornwell will collect the contact details of the staff member/s who will be responsible for organising the 
test delivery in your school. These contact details will include the staff members’ names, school email 
addresses and telephone numbers. 

2. These contact details will be shared with NFER, as NFER will purchase the tests for all the pupils to use. 

3. NFER will liaise with the test provider to ensure delivery of the tests. The test provider in this case is GL 
Assessment (GLA). NFER will share the contact details with GLA. 

4. GLAvwill send information to the nominated staff members about how to administer the test. 

5. The staff member will need to send the specified pupil information from your school management system to 
GLA. This will need to be done in advance of the testing session/s. GLA will provide technical support. 

6. Oncompletion of testing, the staff member will be required to send the test scripts to GLA by special delivery. 


You will need to ensure that there is adequate room provision and support from staff members timetabled 
in to cover the testing session/s. 


Results 


The test will be marked and linked to pupil information by GLA at no cost to your school. Detailed reports are 
downloaded from GLA and can be exported to your school management system. This can be carried out at your 
own convenience. NFER will also have access to these results for the purposes of the evaluation. NFER will 
access these results from the GLA server, so you do not need to send any more data. Pupils’ test responses and 
any other pupil data will be treated with the strictest confidence. The responses will be collected by GL Assessment and 
accessed by NFER. Named data will be matched with the National Pupil Database and shared with EEF. No individual school 
or pupil will be identified in any report arising from the research. 


Responsibilities 


The Bolton project team will: 


e ARRANGE SOUNDS-WRITE AND VEIP TRAINING SESSIONS 

e SUPPLY NECESSARY RESOURCES SPECIFY 

e BE THE FIRST POINT OF CONTACT FOR ANY QUESTIONS ABOUT THE EVALUATION 

e MAKE DECISIONS REGARDING ALLOCATION AND FUNDING 

e PROVIDE ON-GOING TO THE SCHOOL THROUGH REGULAR VISITS BY AN EDUCATIONAL PSYCHOLOGIST 
TO PARTICIPATING SCHOOLS 


The NFER Evaluating Team will: 


e Conduct the random allocation 

e Collect and analyse data from the project collected from the administration of NGRT, the teacher log and 
the process evaluation 

e Ensure all staff carrying out observations and working with pupil data are trained and have received CRB 
clearance 

e Disseminate research findings (the school an pupils’ identities will not be published) 


The School will: 
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e Release 2 English teachers to attend the Sounds Write training course (4 days) and training on the 
Vocabulary Enrichment Intervention Programme (2 days) during the Summer and Autumn Terms 2013. 
This training will be paid for and supply cover provided. 

e Ensure that, where cohort size permits, eligible pupils are split roughly evenly between populations when 
pupils are allocated into forms. 

e Ensure that the Year 7 timetable is arranged so that those students randomly assigned to each 
experimental group can be taught as a discrete withdrawal group and taught by a teacher who has 
attended the Sounds-Write and VEIP training. Funding for additional staffing in the English department 
will be provided for the academic Year 2013-2014 to allow for the creation of these two additional groups 
of pupils. 

e Agree to the Year 7 English curriculum for the pupils in the ‘experimental groups’ being suspended for a 
period of three half terms during the academic Year 2013-2014 to enable the intervention package to be 
delivered. 

e Allow NFER /the project board access to data as necessary for the evaluation of the project. Data 
protection and protocols will be agreed as part of the LA/EEF contract. 

e Allow all necessary testing to be carried out with pupils, which will require access to desks for one lesson 
during the first 2 weeks of September 2013 and one lesson during February 2014. The teachers who 
deliver the VEIP must NOT be involved with administering the tests. 

e Agree to data managers having time to collate details of all eligible pupils and send to GLA during the 
first two weeks of September so that pupils’ data can be linked with the test results. 

e Communicate with parents regarding the project including seeking passive consent for their child to be 
involved in the project, including assessments. 

e Agree to an evaluator from NFER observing one intervention lesson/training sessionif your school is 
chosen. 

e Agree to teachers involved in this project engaging in a semi-structured phone interview with NFER at 
the end of the project. 

e Provide time for teachers to complete the brief ‘teacher logs’ required by NFER. 

e Agree to share attendance data with NFER. 

e Bea point of contact for parents / carers seeking more information on the project. 


We commit to the Bolton EEF Literacy Project as detailed above 


Signatures 


ON BEHALF OF BOLTON LA 
PROJECT MANAGER SUE CORNWELL 


DATE: 


ON BEHALF OF NFER: 


LEAD EVALUATOR BEN STYLES: 


DATE: 14/3/13 


ON BEHALF OF THE SCHOOL: 
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HEAD TEACHER [NAME]: 
OTHER RELEVANT SCHOOLS STAFF [NAMES]: 


DATE: 
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Appendix Ill: Letter to parents 


12" July, 2013. 


Dear Parent/Guardian, 


My name is Sue Cornwell and | am currently co-ordinating a Local Authority project. We recently secured funding 
from the Education Endowment Foundation for a project which will evaluate the educational impact of the 
Vocabulary Enrichment Intervention Programme for Year 7 pupils who could benefit from some additional 
encouragement around reading. This project is running across twelve secondary schools in Bolton and your 
child’s school is participating in this project. | am writing to you to make you aware of what is involved in the 
project and to offer you the chance to raise any questions about it with me directly. Please feel free to contact me 
by email or phone if you have any concerns. | can be contacted via email on susan.cornwell@bolton.gov.uk or by 
telephone on 01204 338060. 


| have attached an information sheet which explains what is involved. We hope that as many pupils as possible 
will be able to participate but we also want to offer you the chance to opt out of the project, if you so wish. All 
children will be asked to sit a reading test in school in September 2013 and February 2014 along with a number of 
their peers. All pupils will then have an equal chance of either accessing the Vocabulary Enrichment Intervention 
Programme in their timetabled English lessons, or accessing the normal English curriculum. If the Vocabulary 
Enrichment Intervention programme is found to be successful in promoting progress in reading it is hoped that 
any children who did not access this programme between September 2013 and February 2014 may have the 
opportunity to access this programme in school from February 2014 onwards. 


If you do not wish your child to take part, please return the reply slip at the bottom of this letter to me 
at the address above as soon as possible and no later than the 2° September 2013. 


If we do not hear from you by this date we will assume that you have no objections and your child will be asked to 
take part. 


Yours faithfully, 


Sue Cornwell 
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Vocabulary Enrichment Intervention Programme 


| do not want my child to be asked to participate in this project. 


Child’s Name: 


School attending in September 2013 


Parent’s Name and Signature: 


Participant Information Sheet 


Study title: 


Evaluating the impact of the Vocabulary Enrichment Intervention Programme with Year 7 pupils 


What is the purpose of the study? 


We wish to evaluate the impact of the Vocabulary Enrichment Intervention Programme on children’s reading 
attainment. This project is being conducted across a number of schools in Bolton. 


Why have we been approached? 


We are looking to support Year 7 pupils who have the potential to achieve more in relation to English attainment 
and reading performance in particular. In particular we are interested in children whose reading achievement 
could be strengthened to enable them to engage more successfully with the secondary school curriculum. 


Do we have to take part? 
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No, participation is entirely voluntary and there are no consequences if you or your child decide not to take part. 


What will happen to my child if s/hetake part? 


If your child takes part, they will be asked to sit a reading test in school in September 2013 and February 2014 
along with a number of their peers. All pupils will then have an equal chance of either accessing the Vocabulary 
Enrichment Intervention Programme in their timetabled English lessons, or accessing the normal English 
curriculum. If the Vocabulary Enrichment Intervention programme is found to be successful in promoting progress 
in reading it is hoped that any children who did not access this programme between September 2013 and 
February 2014 may have the opportunity to access this programme in school from February 2014 onwards. 
However, this is dependent on decisions made by your child’s school. 


What are the possible disadvantages and risks of taking part? 


There are few disadvantages and risks. The reading assessments do take a little time (approximately an hour) to 
complete but we will ensure that they are completed at a time when it will cause minimal disruption to your child’s 
school work. We will do our best to put your child at ease throughout. All results will remain confidential to the 
research team and the school. This project happens during normal lesson time. If the school decides your child 
would benefit from additional school based reading programmes these will still be available. 


What are the possible benefits of taking part? 


We know from previous work in the United States that a similarprogramme appears to boost children’s 
reading. There have also been small scale trials in Britain which have been successful. If we find that it works in 
Bolton, then we anticipate that more schools will offer Vocabulary Enrichment as a reading support method. 


What if something goes wrong? 


You or your child can indicate to the teacher if either of you no longer wish to take part, and you can leave the 
study without question. If you are unhappy with the conduct of the study, you can contact me directly in the first 
instance using the number at the end of this sheet. You are free to withdraw at any point during the study, and for 
up to one month following the completion of the study. You can do this by contacting me and giving me your 
child’s name and the name of his/ her school. 


Will my taking part in this study be kept confidential? 


Yes. Pupils’ test responses and any other pupil data will be treated with the strictest confidence. The 
reading tests will be collected by GL Assessment and accessed by NFER. Named data will be 
matched with the National Pupil Database and shared with the Education Endowment Foundation (a 
government grant funded research foundation), NFER (the UkK’s largest independent provider of 
research, assessment and information services for education) and the UK data archive, allfor research 
purposes. We will not use your child’s name or the name of the school in any report arising from the 
research. 


What will happen io the results of the research study? 
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The reading test data will be used as the basis of a report to be compiled by colleagues from the National 
Foundation for Educational Research, which will be submitted to the Education Endowment Foundation. All 
participating schools will be informed as to the outcomes of the project overall. Teachers will use the information 
from the two reading tests to help establish pupil progress. 


Who is organising and funding the research? 


It is being organised by Bolton Council and is funded by the Education Endowment Foundation. 


Who has reviewed the study? 


The Education Endowment Fund and the National Foundation for Educational Research have reviewed and 
approved this study. 


Contact for Further Information 


Sue Cornwell 


Project Lead 


Tel: 01204 338060 


Email: susan.cornwell@bolton.gov.uk 
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Appendices 


Appendix IV: Security rating summary 


A number of evaluators identified that it would be useful to have a way of summarising the security 
rating given to each report, including the judgement on each of the quality criteria. It is proposed that 
there will be standard summary appendix included in each EEF report. This appendix will clarify what 
the maximum rating a trial could achieve, and how well it was subsequently implemented. This section 
provides a template and three examples from existing EEF trials. 


Security rating summary: Template 


2. Power: 5. Threats to 
1.Design: Samoa 3. Attrition: 4. Balance: Sara a 
ttlek is What is the Adjustment alee ; 
fealialianlelen 2 Adjustment to 


What is the 
uality of the 7-1 eo) me} to rating t 
q E Y fo [-1q-Yeie-] 0) (= seb e beike 2 ee account for issues 

design of the out from the account for : 
effect at the with 


evaluation? WFIU Lale) aig ey-}t-ale=¥a : : 
start? Taie-1ee)e-1e-14 (0) ag 


Evaluation design Implementation Analysis and interpretation 


eee No threats to validity 
Fair and clear experimental 
design (RCT, RDD) 


a 

Posse [ef [I 

= 
| 

i 

an 


The final security rating for this trial is 4 @ . This means that findings are secure. 


The trial was designed as an individually randomised efficacy trial with minimum detectable effect of 
0.16 and could have achieved a maximum of 5 @. The trial experienced 12% attrition which is below 
average and did not result in any observable imbalance on baseline characteristics including prior 
attainment and FSM. However, there was some evidence from the process evaluation of 
contamination as one of the four schools interviewed chose to deliver some aspects of the programme 
to the control group. Also, although the tests were planned to be delivered blind by staff in the 
participating schools not delivering the intervention, it is impossible to guarantee that the staff were 
truly blind to treatment. For this reason the evaluation achieves 4 @. 
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Appendix V: Cost rating 


Cost ratings are based on the approximate cost per pupil of implementing the intervention over one 
year. Cost ratings are awarded using the following criteria. 


Cost Description 

£ Very low: less than £80 per pupil per year. 

££ Low: up to about £170 per pupil per year. 
£ELE Moderate: up to about £700 per pupil per year. 
LLEE High: up to £1,200 per pupil per year. 
Sees Very high: over £1,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission from 
the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk. 
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